Alternate Cutoff Values and DFIT Tests of Measurement Invariance

Likert scales are routinely used in educational and psychological research as measures of constructs of interest. If sound scale development procedures are followed, the resulting scale can reliably and validly measure a construct. However, if a given scale is used to make comparisons among different populations of respondents (e.g., cultures; Riordan & Vandenberg, 1994), over time in longitudinal measurements (Golembiewski, Billingsley, & Yeager, 1976), or across different mediums of data collection (Ployhart, Weekley, Holtz, & Kemp, 2003), measurement invariance must be established before meaningful comparisons in observed data can be made (Raju, Laffitte, & Byrne, 2002; Taris, Bok, & Meijer, 1998; Vandenberg, 2002). Recently, IRT methods of establishing measurement invariance for Likert data have gained acceptance. IRT methods of establishing measurement invariance typically have used the nomenclature of differential item functioning (DIF). DIF is said to occur when the relationship between levels of examinees’ latent trait ( ) and the probability of responses for a particular item differ between two groups (Camilli & Shepard, 1994). The Differential Functioning of Items and Tests (DFIT) framework (Raju, van der Linden, & Fleer, 1995) has been advanced for assessing both DIF and differential test functioning (DTF). Although the DFIT methodology is relatively new, it has been used in several studies published in prestigious journals (e.g., Collins, Raju, & Edwards, 2000; Donovan, Drasgow, & Probst, 2000; Ellis & Mead, 2000; Facteau & Craig, 2001; Flowers, Oshima, & Raju, 1999; Maurer, Raju, & Collins, 1999; Raju, et al., 2002). Articles published in these journals (including Applied Psychological Measurement, Educational and Psychological Measurement, and Journal of Applied Psychology), lend a high level of credibility to the DFIT methodology. Despite its high profile use in the past few years, the DFIT methodology is still under development. Few studies have been published that examine the efficacy of the DFIT program for detecting DIF in Likert data. More importantly, DFIT relies on non-parametric cutoff values for its indices in order to evaluate DIF. The recommended cutoff values have been changed substantially in recent years leading to some questions as to the optimal cutoff value that should be used. In this study, we simulated data with known DIF in order to the efficacy of the DFIT program for detecting DIF using several potentially promising cutoff values.